{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "62a46aad",
   "metadata": {},
   "source": [
    "##### **** The pip install for the transform needs to be adapted to use the appropriate release level. Alternatively, The venv running the jupyter lab could be pre-configured with a requirement file that includes the right release. Example for transform developers working from git clone:\n",
    "```\n",
    "make venv \n",
    "source venv/bin/activate \n",
    "pip install jupyterlab\n",
    "venv/bin/jupyter lab\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ea3570c3",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%capture\n",
    "## This is here as a reference only\n",
    "# Users and application developers must use the right tag for the latest from pypi\n",
    "%pip install 'data-prep-toolkit-transforms[enrichment]'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4297b80e",
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install pandas"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f7b446ec-3dff-4a06-ba49-d22eb0ed165f",
   "metadata": {},
   "source": [
    "##### 1 • Import the transform runtime and utilities"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "915b4c2b-a4d8-469a-8b22-f26156bfd63a",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas\n",
    "from dpk_enrichment.runtime import Enrichment\n",
    "from dpk_enrichment.transform import get_transform_params"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "48dd5577-8e64-4bfa-a33b-c8b545cc3c9d",
   "metadata": {},
   "source": [
    "##### 2 • Required parameters for the transform\n",
    "| Name  | Default Value | Description |\n",
    "|------------|----------|--------------|\n",
    "| **enrichment_content_column_name** | **text** | The column with the content to process. |\n",
    "| **enrichment_lang_column_name** | **lang** | The column name with language identifier for the content. Some of the feature computations require tokenized text as input, the value in this field is used to select the appropriate tokenizer. |\n",
    "| **enrichment_output_column_prefix** | _not set_ | A prefix for the names of all the output columns. Additionally, each column can be explicitly renamed, or if set to an empty string omitted from the output altogether. |\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "556560ca-1596-4dc0-a246-99b0f2ade72b",
   "metadata": {},
   "source": [
    "##### 3 • The parametes to set output column names of the computed features. Refer to [README](./README.md) for the full set of labels."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "9f0cedc3-41f6-4a98-8556-4cd0ae02eb5f",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>parameter</th>\n",
       "      <th>value</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>enrichment_num_newlines_column_name</td>\n",
       "      <td>num_newlines</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>enrichment_num_paragraphs_column_name</td>\n",
       "      <td>num_paragraphs</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>enrichment_num_words_column_name</td>\n",
       "      <td>num_words</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>enrichment_num_chars_column_name</td>\n",
       "      <td>num_chars</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>enrichment_total_non_newline_chars_column_name</td>\n",
       "      <td>total_non_newline_chars</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>enrichment_avg_word_length_column_name</td>\n",
       "      <td>avg_word_length</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>enrichment_avg_paragraph_length_chars_column_name</td>\n",
       "      <td>avg_paragraph_length_chars</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>enrichment_avg_paragraph_length_words_column_name</td>\n",
       "      <td>avg_paragraph_length_words</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>enrichment_alphanumeric_char_ratio_column_name</td>\n",
       "      <td>alphanumeric_char_ratio</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>enrichment_control_char_ratio_column_name</td>\n",
       "      <td>control_char_ratio</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>enrichment_punctuation_char_ratio_column_name</td>\n",
       "      <td>punctuation_char_ratio</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>enrichment_other_symbol_char_ratio_column_name</td>\n",
       "      <td>other_symbol_char_ratio</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>enrichment_tabs_word_ratio_column_name</td>\n",
       "      <td>tabs_word_ratio</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>enrichment_hashes_word_ratio_column_name</td>\n",
       "      <td>hashes_word_ratio</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>enrichment_ellipsis_ratio_column_name</td>\n",
       "      <td>ellipsis_ratio</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>enrichment_bulletpoint_ratio_column_name</td>\n",
       "      <td>bulletpoint_ratio</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>enrichment_dup_paragraphs_ratio_column_name</td>\n",
       "      <td>dup_paragraphs_ratio</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>enrichment_dup_paragraphs_char_ratio_column_name</td>\n",
       "      <td>dup_paragraphs_char_ratio</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>enrichment_top_2_gram_char_ratio_column_name</td>\n",
       "      <td>top_2_gram_char_ratio</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>enrichment_top_3_gram_char_ratio_column_name</td>\n",
       "      <td>top_3_gram_char_ratio</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>enrichment_top_4_gram_char_ratio_column_name</td>\n",
       "      <td>top_4_gram_char_ratio</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>21</th>\n",
       "      <td>enrichment_dup_5_gram_char_ratio_column_name</td>\n",
       "      <td>dup_5_gram_char_ratio</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22</th>\n",
       "      <td>enrichment_dup_6_gram_char_ratio_column_name</td>\n",
       "      <td>dup_6_gram_char_ratio</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23</th>\n",
       "      <td>enrichment_dup_7_gram_char_ratio_column_name</td>\n",
       "      <td>dup_7_gram_char_ratio</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>24</th>\n",
       "      <td>enrichment_dup_8_gram_char_ratio_column_name</td>\n",
       "      <td>dup_8_gram_char_ratio</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25</th>\n",
       "      <td>enrichment_dup_9_gram_char_ratio_column_name</td>\n",
       "      <td>dup_9_gram_char_ratio</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>26</th>\n",
       "      <td>enrichment_dup_10_gram_char_ratio_column_name</td>\n",
       "      <td>dup_10_gram_char_ratio</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                            parameter  \\\n",
       "0                 enrichment_num_newlines_column_name   \n",
       "1               enrichment_num_paragraphs_column_name   \n",
       "2                    enrichment_num_words_column_name   \n",
       "3                    enrichment_num_chars_column_name   \n",
       "4      enrichment_total_non_newline_chars_column_name   \n",
       "5              enrichment_avg_word_length_column_name   \n",
       "6   enrichment_avg_paragraph_length_chars_column_name   \n",
       "7   enrichment_avg_paragraph_length_words_column_name   \n",
       "8      enrichment_alphanumeric_char_ratio_column_name   \n",
       "9           enrichment_control_char_ratio_column_name   \n",
       "10      enrichment_punctuation_char_ratio_column_name   \n",
       "11     enrichment_other_symbol_char_ratio_column_name   \n",
       "12             enrichment_tabs_word_ratio_column_name   \n",
       "13           enrichment_hashes_word_ratio_column_name   \n",
       "14              enrichment_ellipsis_ratio_column_name   \n",
       "15           enrichment_bulletpoint_ratio_column_name   \n",
       "16        enrichment_dup_paragraphs_ratio_column_name   \n",
       "17   enrichment_dup_paragraphs_char_ratio_column_name   \n",
       "18       enrichment_top_2_gram_char_ratio_column_name   \n",
       "19       enrichment_top_3_gram_char_ratio_column_name   \n",
       "20       enrichment_top_4_gram_char_ratio_column_name   \n",
       "21       enrichment_dup_5_gram_char_ratio_column_name   \n",
       "22       enrichment_dup_6_gram_char_ratio_column_name   \n",
       "23       enrichment_dup_7_gram_char_ratio_column_name   \n",
       "24       enrichment_dup_8_gram_char_ratio_column_name   \n",
       "25       enrichment_dup_9_gram_char_ratio_column_name   \n",
       "26      enrichment_dup_10_gram_char_ratio_column_name   \n",
       "\n",
       "                         value  \n",
       "0                 num_newlines  \n",
       "1               num_paragraphs  \n",
       "2                    num_words  \n",
       "3                    num_chars  \n",
       "4      total_non_newline_chars  \n",
       "5              avg_word_length  \n",
       "6   avg_paragraph_length_chars  \n",
       "7   avg_paragraph_length_words  \n",
       "8      alphanumeric_char_ratio  \n",
       "9           control_char_ratio  \n",
       "10      punctuation_char_ratio  \n",
       "11     other_symbol_char_ratio  \n",
       "12             tabs_word_ratio  \n",
       "13           hashes_word_ratio  \n",
       "14              ellipsis_ratio  \n",
       "15           bulletpoint_ratio  \n",
       "16        dup_paragraphs_ratio  \n",
       "17   dup_paragraphs_char_ratio  \n",
       "18       top_2_gram_char_ratio  \n",
       "19       top_3_gram_char_ratio  \n",
       "20       top_4_gram_char_ratio  \n",
       "21       dup_5_gram_char_ratio  \n",
       "22       dup_6_gram_char_ratio  \n",
       "23       dup_7_gram_char_ratio  \n",
       "24       dup_8_gram_char_ratio  \n",
       "25       dup_9_gram_char_ratio  \n",
       "26      dup_10_gram_char_ratio  "
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "params = [(f\"enrichment_{n[0]}\", n[2]) for n in get_transform_params() if n[0] not in [\"content_column_name\", \"lang_column_name\", \"output_column_prefix\"]]\n",
    "pandas.DataFrame(dict(parameter=[p[0] for p in params], value=[p[1] for p in params]))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7234563c-2924-4150-8a31-4aec98c1bf33",
   "metadata": {},
   "source": [
    "##### 4 • Setup the transform"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "badafb96-64d2-4bb8-9f3e-b23713fd5c3f",
   "metadata": {},
   "outputs": [],
   "source": [
    "transform = Enrichment(\n",
    "    enrichment_content_column_name=\"text\",\n",
    "    enrichment_lang_column_name=\"lang\",\n",
    "    enrichment_output_column_prefix=\"e_\",\n",
    "    input_folder=\"test-data/input\", \n",
    "    output_folder=\"output\" \n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "853aba4b-0094-4264-8677-5371f1141142",
   "metadata": {},
   "source": [
    "##### 5 • Run"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "930748f2-f7f5-459c-b2e7-8e3c54873838",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "12:32:34 INFO - Enrichment parameters are: {\"output_column_prefix\": \"e_\", \"content_column_name\": \"text\", \"lang_column_name\": \"lang\", \"num_newlines_column_name\": \"num_newlines\", \"num_paragraphs_column_name\": \"num_paragraphs\", \"num_words_column_name\": \"num_words\", \"num_chars_column_name\": \"num_chars\", \"total_non_newline_chars_column_name\": \"total_non_newline_chars\", \"avg_word_length_column_name\": \"avg_word_length\", \"avg_paragraph_length_chars_column_name\": \"avg_paragraph_length_chars\", \"avg_paragraph_length_words_column_name\": \"avg_paragraph_length_words\", \"alphanumeric_char_ratio_column_name\": \"alphanumeric_char_ratio\", \"control_char_ratio_column_name\": \"control_char_ratio\", \"punctuation_char_ratio_column_name\": \"punctuation_char_ratio\", \"other_symbol_char_ratio_column_name\": \"other_symbol_char_ratio\", \"tabs_word_ratio_column_name\": \"tabs_word_ratio\", \"hashes_word_ratio_column_name\": \"hashes_word_ratio\", \"ellipsis_ratio_column_name\": \"ellipsis_ratio\", \"bulletpoint_ratio_column_name\": \"bulletpoint_ratio\", \"dup_paragraphs_ratio_column_name\": \"dup_paragraphs_ratio\", \"dup_paragraphs_char_ratio_column_name\": \"dup_paragraphs_char_ratio\", \"top_2_gram_char_ratio_column_name\": \"top_2_gram_char_ratio\", \"top_3_gram_char_ratio_column_name\": \"top_3_gram_char_ratio\", \"top_4_gram_char_ratio_column_name\": \"top_4_gram_char_ratio\", \"dup_5_gram_char_ratio_column_name\": \"dup_5_gram_char_ratio\", \"dup_6_gram_char_ratio_column_name\": \"dup_6_gram_char_ratio\", \"dup_7_gram_char_ratio_column_name\": \"dup_7_gram_char_ratio\", \"dup_8_gram_char_ratio_column_name\": \"dup_8_gram_char_ratio\", \"dup_9_gram_char_ratio_column_name\": \"dup_9_gram_char_ratio\", \"dup_10_gram_char_ratio_column_name\": \"dup_10_gram_char_ratio\"}\n",
      "12:32:34 INFO - pipeline id pipeline_id\n",
      "12:32:34 INFO - code location None\n",
      "12:32:34 INFO - data factory data_ is using local data access: input_folder - test-data/input output_folder - output\n",
      "12:32:34 INFO - data factory data_ max_files -1, n_sample -1\n",
      "12:32:34 INFO - data factory data_ Not using data sets, checkpointing False, max files -1, random samples -1, files to use ['.parquet'], files to checkpoint ['.parquet']\n",
      "12:32:34 INFO - orchestrator enrichment started at 2025-03-12 12:32:34\n",
      "12:32:34 INFO - Number of files is 2, source profile {'max_file_size': 0.21838951110839844, 'min_file_size': 0.1921825408935547, 'total_file_size': 0.4105720520019531}\n",
      "[nltk_data] Downloading package punkt_tab to\n",
      "[nltk_data]     /Users/cpendus/nltk_data...\n",
      "[nltk_data]   Package punkt_tab is already up-to-date!\n",
      "12:32:35 INFO - Completed 1 files (50.0%) in 0.014 min\n",
      "12:32:37 INFO - Completed 2 files (100.0%) in 0.037 min\n",
      "12:32:37 INFO - Done processing 2 files, waiting for flush() completion.\n",
      "12:32:37 INFO - done flushing in 0.0 sec\n",
      "12:32:37 INFO - Completed execution in 0.037 min, execution result 0\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "0"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "transform.transform()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c3df5adf-4717-4a03-864d-9151cd3f134b",
   "metadata": {},
   "source": [
    "##### 6 • Show the output with the computed features"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "845a75cf-f4a9-467d-87fa-ccbac1c9beb8",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>text</th>\n",
       "      <th>lang</th>\n",
       "      <th>e_lid_score</th>\n",
       "      <th>e_num_newlines</th>\n",
       "      <th>e_num_paragraphs</th>\n",
       "      <th>e_num_words</th>\n",
       "      <th>e_num_chars</th>\n",
       "      <th>e_total_non_newline_chars</th>\n",
       "      <th>e_avg_word_length</th>\n",
       "      <th>e_avg_paragraph_length_chars</th>\n",
       "      <th>...</th>\n",
       "      <th>e_dup_paragraphs_char_ratio</th>\n",
       "      <th>e_top_2_gram_char_ratio</th>\n",
       "      <th>e_top_3_gram_char_ratio</th>\n",
       "      <th>e_top_4_gram_char_ratio</th>\n",
       "      <th>e_dup_5_gram_char_ratio</th>\n",
       "      <th>e_dup_6_gram_char_ratio</th>\n",
       "      <th>e_dup_7_gram_char_ratio</th>\n",
       "      <th>e_dup_8_gram_char_ratio</th>\n",
       "      <th>e_dup_9_gram_char_ratio</th>\n",
       "      <th>e_dup_10_gram_char_ratio</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Alisha // Twenty two // California\\nThis is my...</td>\n",
       "      <td>en</td>\n",
       "      <td>0.999</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>565</td>\n",
       "      <td>2467</td>\n",
       "      <td>2467</td>\n",
       "      <td>3.534513</td>\n",
       "      <td>2467.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.016026</td>\n",
       "      <td>0.019231</td>\n",
       "      <td>0.025641</td>\n",
       "      <td>0.005342</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>December 2020 – Low Interest Payday Loans KZDO...</td>\n",
       "      <td>en</td>\n",
       "      <td>0.998</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>395</td>\n",
       "      <td>2100</td>\n",
       "      <td>2100</td>\n",
       "      <td>4.440506</td>\n",
       "      <td>2100.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.023795</td>\n",
       "      <td>0.045211</td>\n",
       "      <td>0.045211</td>\n",
       "      <td>0.084474</td>\n",
       "      <td>0.086258</td>\n",
       "      <td>0.097561</td>\n",
       "      <td>0.088638</td>\n",
       "      <td>0.056514</td>\n",
       "      <td>0.035098</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>terminology com share|improve this question ed...</td>\n",
       "      <td>en</td>\n",
       "      <td>0.999</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>290</td>\n",
       "      <td>1570</td>\n",
       "      <td>1570</td>\n",
       "      <td>4.565517</td>\n",
       "      <td>1570.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.012698</td>\n",
       "      <td>0.031746</td>\n",
       "      <td>0.019048</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Wilmore Road Nicholasville, KY home and proper...</td>\n",
       "      <td>en</td>\n",
       "      <td>0.966</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>660</td>\n",
       "      <td>4043</td>\n",
       "      <td>4043</td>\n",
       "      <td>5.259091</td>\n",
       "      <td>4043.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.320049</td>\n",
       "      <td>0.454323</td>\n",
       "      <td>0.291845</td>\n",
       "      <td>0.455855</td>\n",
       "      <td>0.371551</td>\n",
       "      <td>0.172900</td>\n",
       "      <td>0.207235</td>\n",
       "      <td>0.224402</td>\n",
       "      <td>0.237278</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>'I went from a screaming room of people to dea...</td>\n",
       "      <td>en</td>\n",
       "      <td>0.999</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>281</td>\n",
       "      <td>1594</td>\n",
       "      <td>1594</td>\n",
       "      <td>4.754448</td>\n",
       "      <td>1594.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.009288</td>\n",
       "      <td>0.008514</td>\n",
       "      <td>0.010062</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>95</th>\n",
       "      <td>Khubz (Traditional Iraqi Bread) - shosholuv\\nK...</td>\n",
       "      <td>en</td>\n",
       "      <td>0.999</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>534</td>\n",
       "      <td>2726</td>\n",
       "      <td>2726</td>\n",
       "      <td>4.211610</td>\n",
       "      <td>2726.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.020737</td>\n",
       "      <td>0.011982</td>\n",
       "      <td>0.013364</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>96</th>\n",
       "      <td>Property tax payments must be federally post m...</td>\n",
       "      <td>en</td>\n",
       "      <td>0.999</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>216</td>\n",
       "      <td>1121</td>\n",
       "      <td>1121</td>\n",
       "      <td>4.324074</td>\n",
       "      <td>1121.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.013514</td>\n",
       "      <td>0.036036</td>\n",
       "      <td>0.040541</td>\n",
       "      <td>0.023649</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>97</th>\n",
       "      <td>All About Bail Bonds Offers Assistance of Expe...</td>\n",
       "      <td>en</td>\n",
       "      <td>0.999</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>437</td>\n",
       "      <td>2369</td>\n",
       "      <td>2369</td>\n",
       "      <td>4.556064</td>\n",
       "      <td>2369.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.057351</td>\n",
       "      <td>0.066736</td>\n",
       "      <td>0.052138</td>\n",
       "      <td>0.058916</td>\n",
       "      <td>0.049531</td>\n",
       "      <td>0.040146</td>\n",
       "      <td>0.021898</td>\n",
       "      <td>0.026069</td>\n",
       "      <td>0.027112</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>98</th>\n",
       "      <td>Windies stuttering after conceding a massive l...</td>\n",
       "      <td>en</td>\n",
       "      <td>0.999</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>967</td>\n",
       "      <td>4974</td>\n",
       "      <td>4974</td>\n",
       "      <td>4.199586</td>\n",
       "      <td>4974.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.012030</td>\n",
       "      <td>0.019549</td>\n",
       "      <td>0.021053</td>\n",
       "      <td>0.016040</td>\n",
       "      <td>0.009774</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>99</th>\n",
       "      <td>13 Tips How to Reduce Double Chin &amp; Face Fat F...</td>\n",
       "      <td>en</td>\n",
       "      <td>0.998</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>351</td>\n",
       "      <td>1763</td>\n",
       "      <td>1763</td>\n",
       "      <td>4.128205</td>\n",
       "      <td>1763.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.055596</td>\n",
       "      <td>0.023827</td>\n",
       "      <td>0.034657</td>\n",
       "      <td>0.013718</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>100 rows × 30 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                 text lang  e_lid_score  \\\n",
       "0   Alisha // Twenty two // California\\nThis is my...   en        0.999   \n",
       "1   December 2020 – Low Interest Payday Loans KZDO...   en        0.998   \n",
       "2   terminology com share|improve this question ed...   en        0.999   \n",
       "3   Wilmore Road Nicholasville, KY home and proper...   en        0.966   \n",
       "4   'I went from a screaming room of people to dea...   en        0.999   \n",
       "..                                                ...  ...          ...   \n",
       "95  Khubz (Traditional Iraqi Bread) - shosholuv\\nK...   en        0.999   \n",
       "96  Property tax payments must be federally post m...   en        0.999   \n",
       "97  All About Bail Bonds Offers Assistance of Expe...   en        0.999   \n",
       "98  Windies stuttering after conceding a massive l...   en        0.999   \n",
       "99  13 Tips How to Reduce Double Chin & Face Fat F...   en        0.998   \n",
       "\n",
       "    e_num_newlines  e_num_paragraphs  e_num_words  e_num_chars  \\\n",
       "0                0                 1          565         2467   \n",
       "1                0                 1          395         2100   \n",
       "2                0                 1          290         1570   \n",
       "3                0                 1          660         4043   \n",
       "4                0                 1          281         1594   \n",
       "..             ...               ...          ...          ...   \n",
       "95               0                 1          534         2726   \n",
       "96               0                 1          216         1121   \n",
       "97               0                 1          437         2369   \n",
       "98               0                 1          967         4974   \n",
       "99               0                 1          351         1763   \n",
       "\n",
       "    e_total_non_newline_chars  e_avg_word_length  \\\n",
       "0                        2467           3.534513   \n",
       "1                        2100           4.440506   \n",
       "2                        1570           4.565517   \n",
       "3                        4043           5.259091   \n",
       "4                        1594           4.754448   \n",
       "..                        ...                ...   \n",
       "95                       2726           4.211610   \n",
       "96                       1121           4.324074   \n",
       "97                       2369           4.556064   \n",
       "98                       4974           4.199586   \n",
       "99                       1763           4.128205   \n",
       "\n",
       "    e_avg_paragraph_length_chars  ...  e_dup_paragraphs_char_ratio  \\\n",
       "0                         2467.0  ...                          0.0   \n",
       "1                         2100.0  ...                          0.0   \n",
       "2                         1570.0  ...                          0.0   \n",
       "3                         4043.0  ...                          0.0   \n",
       "4                         1594.0  ...                          0.0   \n",
       "..                           ...  ...                          ...   \n",
       "95                        2726.0  ...                          0.0   \n",
       "96                        1121.0  ...                          0.0   \n",
       "97                        2369.0  ...                          0.0   \n",
       "98                        4974.0  ...                          0.0   \n",
       "99                        1763.0  ...                          0.0   \n",
       "\n",
       "    e_top_2_gram_char_ratio  e_top_3_gram_char_ratio  e_top_4_gram_char_ratio  \\\n",
       "0                  0.016026                 0.019231                 0.025641   \n",
       "1                  0.023795                 0.045211                 0.045211   \n",
       "2                  0.012698                 0.031746                 0.019048   \n",
       "3                  0.320049                 0.454323                 0.291845   \n",
       "4                  0.009288                 0.008514                 0.010062   \n",
       "..                      ...                      ...                      ...   \n",
       "95                 0.020737                 0.011982                 0.013364   \n",
       "96                 0.013514                 0.036036                 0.040541   \n",
       "97                 0.057351                 0.066736                 0.052138   \n",
       "98                 0.012030                 0.019549                 0.021053   \n",
       "99                 0.055596                 0.023827                 0.034657   \n",
       "\n",
       "    e_dup_5_gram_char_ratio  e_dup_6_gram_char_ratio  e_dup_7_gram_char_ratio  \\\n",
       "0                  0.005342                 0.000000                 0.000000   \n",
       "1                  0.084474                 0.086258                 0.097561   \n",
       "2                  0.000000                 0.000000                 0.000000   \n",
       "3                  0.455855                 0.371551                 0.172900   \n",
       "4                  0.000000                 0.000000                 0.000000   \n",
       "..                      ...                      ...                      ...   \n",
       "95                 0.000000                 0.000000                 0.000000   \n",
       "96                 0.023649                 0.000000                 0.000000   \n",
       "97                 0.058916                 0.049531                 0.040146   \n",
       "98                 0.016040                 0.009774                 0.000000   \n",
       "99                 0.013718                 0.000000                 0.000000   \n",
       "\n",
       "    e_dup_8_gram_char_ratio  e_dup_9_gram_char_ratio  e_dup_10_gram_char_ratio  \n",
       "0                  0.000000                 0.000000                  0.000000  \n",
       "1                  0.088638                 0.056514                  0.035098  \n",
       "2                  0.000000                 0.000000                  0.000000  \n",
       "3                  0.207235                 0.224402                  0.237278  \n",
       "4                  0.000000                 0.000000                  0.000000  \n",
       "..                      ...                      ...                       ...  \n",
       "95                 0.000000                 0.000000                  0.000000  \n",
       "96                 0.000000                 0.000000                  0.000000  \n",
       "97                 0.021898                 0.026069                  0.027112  \n",
       "98                 0.000000                 0.000000                  0.000000  \n",
       "99                 0.000000                 0.000000                  0.000000  \n",
       "\n",
       "[100 rows x 30 columns]"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pandas.read_parquet('output/1.parquet', engine='pyarrow')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7aef6ac9-96cf-40ad-a472-b5d9036436e5",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
